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Documenting Evaluation Use: 
Guided Evaluation Decisionmaking 



Report Prepared By 
James Burry 



Introduction 



During the 1984-85 school year, selected districts throughout the 
country have been applying our framework for evaluation use to increase the 
utilization of their program evaluations. In this endeavor, district 
administrators have been applying the organizing framework and 
use-promoting procedures contained in our Guide for Evaluation Decision 
Makers {Alkln et al, 1985). 

The Guide, which characterizes the factors our research Identified as 
affecting evaluation use, acts upon the premise that if evaluations are to 
have a potential for use, they need to be organized around the factors 
known to affect use. 

In this regard, our research found that it is often possible to 
influence these factors in the interest of use. Many of these factors, 
however, are not particularly amenable to evaluator influence. Rather, 
they are often part of an administrators 's domain of responsibility, and 
therefore more amenable to administrator influence. 

Given the strength of this finding, and because much of the emphasis 
on evaluation improvement focuses on evaluator responsibilities, we set out 
to develop a tool which would improve practice by assisting administrators 
tc organize evaluations for use. The Guide we developed to meet this need 
discusses the administrator's organizing role and offers procedures for 
promoting and monitoring the use process. 

This paper documents the use process among our participant districts 
and demonstrates the successes they achieved. First, we discuss the 
research that led to our conclusions concerning the administrator's role in 
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evaluation use. Second, we characterize the objectives of the project and 
how they grew out of our research findings. Third, we describe the process 
of locating and selecting districts to implement the Guide. Fourth, we 
describe each evaluation context, the use-affecting factors which emerged 
at each site, how administrators applied the Guide to promote use, and how 
well the intended evaluation uses were achieved. Fifth, we discuss some 
implementation issues. Finally, we synthesize the results across sites and 
suggest implications for future practice. 



CSE Research on Evaluation Use 



Our research on evaluation use shows clearly that the more active an 
administrators participation in program evaluation, the more likely it is 
that the evaluation findings will be put to use. Given the strength of 
this finding, we developed an organizing framework to help administrators 
take the leadership role necessary to enhance their program evaluations 1 
use potential. The framework takes the position that evaluation is a means 
of producing information which can be used to make decisions about 
programs: about how the prorram can be improved, about how resources might 
be allocated, about how attitudes toward the program can be changed. 

Now, the people who will be asked to use evaluation information for 
the above and other decision areas are likely to differ in their 
disposition to apply information. In fact, predisposition toward 
evaluation is one of the many elements known to influence use, elements 
inherent not only in the conduct of the evaluation itself but also in its 
social, political, organizational, administrative, and programmatic 



context. If these elements are Ignored, they can reduce or negate the 
evaluation's use potential. 

If, on the other hand, an evaluation is considered in light of the 
range of elements known to affect use — and if it is organized, planned, 
conducted, and communicated in response to the particular elements 
operating in a given setting — its use potential increases. 

As we mentioned previously, we believe that a program administrator, 
rather than a program evaluator, is frequently in the strongest strategic 
position to assume the evaluation organizing responsibility. 

In large measure, our research on evaluation use over the past decade 
was motivated by the grandiose claims made by some of the early advocates 
of evaluation. According to them, evaluation would have a significant and 
highly visible Impact on the program being evaluated. All the evaluator 
had to do was to provide valid data and educators would use them in their 
decision-making which would result in dramatic improvements in educational 
policy and practice. Such direct and immediate use 1s rare. 

However, we have discovered over the years that although info ation 
may not be put to Immediate and observable use, that should not be taken to 
suggest that no use is taking place. Rather, an evaluation's impact is 
frequently long-range, cumulative, and more modest than originally expected 
(Alkin et al., 1974). Further, other kinds of Information, in addition to 
evaluation, often contribute to a particular decision. 

In addition, the early proponents of evaluation frequently placed toe 
much emphasis on the technical aspects of evaluation. Once again, our 
research offers a different perspective. That is, neither the procedural 
soundness of an evaluation nor the validity of its data are sufficient to 
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ensure use. For example, one early CSE study (Alkin, 1975) showed that the 
evaluator's stance toward the program, its staff, and their questions can 
affect use. Other early research (Patton et al., 1978) also pointed up the 
importance of the "personal factor." 

Among the early evaluation researchers, further, the evaluator's 
responsibility for evaluation and its use was overplayed. Our research, 
however, indicated that the potential user of information, such as an 
administrator, also carries some responsibility. The evaluator often lacks 
the power, prestige, understanding, and political sensitivity necessary to 
promote use. The influence of an administrator is often required. 

Drawing on the early studies mentioned above, CSE launched a series of 
empirical studies of evaluation use. They included evaluation case 
studies, an evaluator field study, and a user survey. The findings from 
these studies prompted us to review the literature on evaluation use, which 
contributed to our development of the Guide for evaluation decision- 
makers. The major points of each of these studies — their contributions 
to our knowledge of the evaluation use process ~ are summarized below. 
The Evaluation Case Studies 

Our case studies (Alkin, Daillak, & White, 1979) focused, over a 
period of two years, on five Title I or Title IVC programs. These studies 
involved in-depth interviews with program staff and the evaluator, and the 
examination of documents such as program proposals and evaluation reports. 

The case studies described in detail school-level program evaluation: 
how the evaluation process unfolded and who helped shape the process, how 
evaluation was used, how it fitted in with other school operations, and how 
it influenced decisions about the program. By identifying the factors that 
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influenced the evaluation process, we developed a conceptional framework to 
guide our future study of use. This early framework identified the 
evaluation's constraints, the evaluator's approach to the program, the 
user's disposition toward evaluation, the organizational setting, and the 
administrator's leadership as influencing the use process. 
The Evaluator Field Study 

The case studies showed that evaluation information is more likely to 
be used at the local level when evaluators adapt their approach to the 
needs of program managers. The field study was therefore conducted to 
examine the evaluation process from the standpoint of evaluators 
themselves. Daillak (ig80) spent a year in an urban school system, working 
closely with three district staff evaluators. By directly observing these 
three evaluators at work, and by discussing with them the evaluation 
process as it unfolded, Daillak illuminated some of the elements posited in 
the original version of the conceptual framework, especially those elements 
related to evaluator approach, and hew he or she communicates information 
in light of the evaluation's organizational setting. 

Witn respect to approach, for instance, Daillak observed the 
importance of the evaluator's providing advice and guidance to program 
staff rather than merely collecting and analyzing data. In terms of 
information communication, he found that the informal sharing of ideas and 
recommendations tended to increase evaluation use. With respect to the 
organizational setting, Daillak found that, regardless of the approach 
taken by the evaluator, certain organizational constraints — such as loose 
management of instruction or negative attitudes toward evaluation — may 
pose obstacles to program evaluation and information use. 
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The User Survey 

The user survey (Stecher, Alkin, & Flesher, 1981), conducted in the 
same district as tne evaluator field study, was intended to shed light on 
the evaluation process as seen by users: how they view the program and its 
evaluation, how evaluation information i* used and by whom, how much it is 
used, for what purposes, and under what social/institutional/political 
conditions. Over the course of a year, the principal, the program 
coordinator, and another staff administrator at each of 22 schools were 
interviewed about their perceptions of significant program occurrences over 
the year and about the extent to which evaluation had contributed to these 
occurrences. 

The interviews suggested that influences beyond evaluation, such as 
personal belief and opinion, affect the decisionmaking process. Further, 
while evaluation may play a modest role in the final decision, its ? mpact 
is strongly felt in the earlier stages of the decisionmaking process, such 
as needs assessment and problem recognition. Finally, evaluation use 
increases to the extent that administrators adopt the tactic of involving 
other potential users, such as teachers, in the decisionmaking process. 
Review of the Literature on Evaluation Use 

At this stage, we had fleshed out our framework for evaluation use. 
We had a fairly clear picture of the general conditions of evaluation use, 
and had identified many specific elements — such as evaluator role, user 
interest, the evaluation requirements, and the substance of the evaluation 
information — that have a bearing on use in most settings. 

We conducted a review of the evaluation use literature (Burry, 1983), 
to judge the extent to which our findings might be shared by others. The 
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review identified and examined some 150 studies dealing specifically with 
evaluation use: in education, public health, human services, and criminal 
justice. These studies, em; irlcal as well as conceptual or theoretical, 
confirmed the existence and Importance of each of the elements identified 
earlier. They also made it clear, just as we had found in our work, that 
evaluation use is strongly influenced by the actions of administrators: 
identifying the intended information uurs, translating cheir interests 
into evaluation questions, discussing these and other requirements with the 
evaluator, making sure that the evaluation addresses both program-focused 
and other Issues, ?nd making sure that the answers are communicated in ways 
appropriate to the Intended users. 

The Guide for Evaluation Decision-makers 

In vietf of the importance of adminlstrator-evaluator collaboration in 
mapping out the evaluation's focus and procedures, we developed our Guide 
for administrators seeking to organize their evaluations for maximal 
usefulness (Alkin et al., 1985). The Guide exemplifies the personal, 
organizational, and procedural characteristics known to affect use. Of 
major importance to the administrator- organizer are scenarios which offer a 
bridge between the research findings and their administrative application. 
These scenarios, a series of evolving vignettes, show an administrator 
working closely with an evaluator to build local uses into the evaluation. 

The organizing framework presented in the Guide demonstrates how the 
elements affecting use fall into patterns which reflect stages of the use 
process and how the administrator can influence these elements to promote 
use. 



Over the last decade, then, our research on evaluation use has evolved 
from investigation of the general conditions of use to analysis of specific 
cases. In this evolution, we have moved from an understanding of the broad 
domain of use, to consideration ano verification of the discrete elements 
that make up that domain, to illumination of how these elements influence 
the use process, and finally to demonstration of the kinds of administrator 
action which promote evaluation use. 

Figure 1, which is excerpted from the Guide, shows that the principal 
characteristics influencing use fall into three categories: the 
participants in the evaluation, the setting of the evaluation, and the 
conduct of the evaluation. The first category comprises ^valuator 
characteristics and user characteristics; the second, preexisting 
evaluation bounds, organizational features, and program characteristics; 
and the third, evaluation procedures, information dialogue, substance of 
the evaluation, and reporting of the evaluation. Each of the elements 
listed in the figure has a demonstrated relevance to evaluation use. 
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Figure 1: The Domain of Evaluation Use 



I, The Participants in the Evaluation 

A. Evaluator Characteristics 

1. commitment to use 

2. willingness to involve users 

3. choice of role 

4. rapport with users 

5. political sensitivity 

6. credibility 

B . Us er Characteristics 

1. identity 

a. range of potential users 

b. organizational positions 

c. professional experience levels 

2. interest in the evaluation 

a. views about the project being evaluated 

b. expectations for the evaluation 

c. predisposition toward the evaluation 

d. perceived need 

e. perceived risks 

3. commitment to use 

4. professional style 

a. administrative and organizational skills 

b. initiative 

c. openness to new ideas or change 

5. information processing 

a. preference for particular forms 

b. how information is p ( ocessed 
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I K The Setting of the Evaluation 

A. Pre-existing Evaluation Bounds 

1. written requirements 

2. other contractual obligations 

3. fiscal constraints 

B. Organizational Features 

1. iritraorganizational 

a. role of central/district office 

b. interrelationship between unit and central/district 
adnrini stratlon 

c. Institutional arrangements 

d. unit level autonomy 

e. sources of information beyond evaluation likely to 
be in use 

f. perceived institutional risk 

2. external 

a. community climate 

b. community influence 

c. role of other agencies 

C. Project Characteristics 

1. age/maturity 

2. innovati veness 

3. overlap with other projects 

III. The Conduct of the Evaluation 
A. Evaluation Procedures 

1. methods used 

a. appropriateness 

b. rigor 

2. dealing with mandated tasks 

3. use of a general model 
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B. Information Dialogue 

1. amount and quality of interaction between evaluator and 
users 

C. Substance of Evaluation Information 

1. information relevance 

2. information specificity 

D. Evaluation Reporting 

1. frequency of information provided 

2. timing of information 

3. format of presentations 

a. oral presentations 

b. written reports 

c. statistical and narrative data 
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Summary: How the Guide Distills the Research Findings 

In this section we discuss how the Guide distills the relevant 
rc xh for administrator application. A detailed synthesis of the 
evaluation-use research appears in Burry, Alkin, & Ruskus, 1985. 

Evaluator characteristics affecting use : As Figure 1 shows, the 
evaluatcr characteristics affecting use are: commitment to use, 
willingness to involve users, choice of role, rapport with users, political 
sensitivity, and credibility. All these characteristics reflect the 
^valuator's approach to his or her craft, which has a profound influence on 
use. 

For instance, when evaluators tangibly demonstrate their commitment to 
use and encourage others to follow suit, use is more likely to occur. For 
example, some evaluators attempt to stimulate use of findings by involving 
users in the evaluation itself. Some evaluators committed to use often 
successfully involve others in the evaluation by enlisting the support of 
an administrator, who then goes on to recruit others. 

Evaluator commitment to use and willingness to involve others, then, 
often go hand in hand. But these qualities may have little effect if the 
evaluator does not adopt a role appropriate to the program being 
evaluated. In most cases, the most appropriate role seems to be a 
facilitative one, based on a recognition that the evaluation should serve 
the program ratner than simply meeting the pro forma — frequently external 
— requirements. 

The ability to assume a facilitative role depends on the extent to 
which the evaluator is willing to collaborate with potential users, to 
involve them in decisions about the evaluation, and then to ensure that 
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information gets to people who can act upon it. 

An evaluator's ability to adopt a collaborative role may influence the 
extent to which he or she can develop rapport with users. And such rapport 
contributes greatly to use. Further, both rapport and use increase when 
the evaluator demonstrates an understanding of the internal and external 
political environment of the program. In our framework, this 
characteristic is termed "political sensitivity," the understanding that 
evaluation is only one part of the political context of any program. 

The final evaluator characteristic contributing to use is credibility 
with the program staff. Credibility may stem from reputation, from the 
trust the evaluator enjoys with program staff and from their perception of 
how well he or she understands their program, or from forcefulness of 
personality. 

As with any profession, evaluators differ in their approach to their 
craft. Some evaluators believe that technical soundness leads to 
information use. Others understand that their personal commitment to use 
is critical and that they must expend whatever effort is needed to promote 
use. Efforts to involve potential users in all facets of the evaluation 
process are especially likely to pay off. 

These two characteristics — commitment to use and willingness to 
involve others — depend, in large part, on the role the evaluator chooses, 
and there are several from which to choose. Some evaluators view 
themselves as impartial judges; others equate evaluation with research and 
run the evaluation like an experiment; others act as advocates for the 
program being evaluated. The ideal role, however, is one in which the 
evaluator views the users as colleagues who help guide the evaluation to 
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ensure that their questions are built in and answered. 

The evaluator who adopt the use-promoting stance suggested above 
takes an important step toward fostering the trust and harmony that underly 
rapport with users, a rapport that is further strengthened when the 
evaluator is sensitive to the program's political dynamics and understands 
that evaluation information is only one of many possible inputs to the 
decisionmaking process and that people with different attitudes, 
backgrounds, and power or prestige are like'y to contribute to that 
process. 

Finally, the evaluator who is a good craftsman and who displays the 
characteristics described above is establishing the credibility he or she 
needs if potential users are to believe that the information they receive 
is worth putting to use. 

User characteristics affecting use : The user characteristics 
pertinent to use are: identity, interest in the evaluation, commitment to 
use, professional style, and -'nformation preferences and processing 
routines. 

Most evaluations have a range of potential users; evaluation use tends 
to increase when these users are clearly specified and when their questions 
and needs are earmarked for the evaluator 's attention. The use-conscious 
evaluator, however, should be aware that users, even those who belong to 
the same group, may have conflicting definitions of a common issue and that 
these conflicts must be resolved in the interest of use. 

Other user characteristics included in identity are organizational 
position(s) and professional experience level (s). In this regard, use 
tends to increase when evaluation information is provided to someone who 
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occupies a position of power within the organization and who has sufficient 
experience to put the information to use. 

User interest has several aspects: users 1 views about the program 
being evaluated, their expectations about the evaluation, their 
predisposition toward evaluation in general, their perceived needs, and 
their perceptions of the risks entailed in an evaluation. Successful 
evaluators are careful to determine how the various users view both the 
program and thf valuation and then act accordingly. For example, if users 
hold inflexible views about a program, if they expect the evaluation to 
produce information that supports their views, and if the evaluation 
generates findings that run counter to these views, then use is difficult 
to promote. 

Users 1 general attitudes toward evaluation are also important: the 
more positive these attitudes, the more likely it is that the evaluation 
will be used. For instance, use increases when the potential users believe 
that evaluation produces useful information and is appropriate for their 
particular questions. However, when users believe that the evaluation 
entails a risk that outweighs its perceived benefits, use levels may 
diminish. 

User commitment to use is just as important as evaluator commitment. 
For instance, when decisionmakers are tangibly and visibly committed to 
use, then use potential increases. 

Users' professional styles constitute another important group of 
influences. In this regard, the administrator's ability to organize, 
initiate, and follow through on information-based action is related to 
use. Similarly, the extent to which management and evaluation functions 
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are Integra wed has a strong bearing on use. Further, administrators who 
are open to change* and who have the authority and initiative to apply the 
information at hand, have a positive influence on use. 

Finally, users differ not only in their preferences for particular 
kinds of inT jn but alsc in their preferences as to how this 
information is processed and presented to them. The extent to which 
evaluation information matches these preferences is a principal determinant 
of use. 

Given that most evaluations have a range of potential users, it 
follows that various groups of users are likely to differ on the 
characteristics outlined above. Evaluators and administrators concerned 
v;ith use rr 1 be aware of these differences and adapt their approach 
accordingly. 

Users 1 views about the program, its evaluation, and evaluation in 
general may range from neutral to strongly positive or negative, and both 
the nature and the strength of th*3e views can affect their tendency to use 
evaluation information. Generally, to the extent that users 1 specific 
needs for evaluation information outweigh the perceived risks involved in 
the evaluation, they will be inclined to put the information to use. 
Actual use, however, will depend on how well users can organize information 
and cake responsibility for putting it to use, even if such use leads to 
changes in the program's content and operation. 

Finally, no matter how strongly users are committed to applying 
evaluation information, use will also depend on the extent to vhich the 
information they receive comes to them in a form which they are 
comfortable. 
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Chara cteristics of the setting affecting use : The features of the 
setting that are pertinent to use include: preexisting evaluation bounds, 
organizational features, and program characteristics. 

Evaluation bounds have three facets: written evaluation requirements, 
other contractual obligations, and fiscal constraints. Any potential 
conflicts (e.g., between the requirements imposed on an evaluation by an 
outside agency and the requirements emanating from the program itself) must 
be resolved if use is to occur; no single set of issues should be allowed 
to dominate. For example, use increases when the parties associated with 
different requirements agree in advance about the focus of the evaluation 
and the kinds of information it should produce. Negotiation between the 
evaluator and various users also contributes to the use process, as does 
discussion of the extent to which financial and personal resources 
earmarked for an evaluation are allocated to program concerns. 

The organizational features affecting use are of two kinds: 
intraorganizational and external. With respect to intraorganizational 
features, the use potential tends to increase when a district office gives 
the evaluator some freedom in choosing a role and an evaluation focus. 
Equally important is the district's overall responsiveness to the needs of 
the local program. In particular, local program personnel must have 
autonomy to act upon the evaluation's findings. Allowing the evaluator 
freedom in role selection without offering comparable freedom at the site 
level to follow the evaluator's recommendations runs counter to the use 
process. 

Just as individual users may perceive that an evaluation entails risks 
to a program, so the organization as a whole may perceive risk, and this 
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perception can impede use. In this regard, the more that users believe 
there is a risk to the institution, the more they should be involved in 
decisions that may affect the institution. Given the importance of this 
involvement to the use process, it is critical that overall institutional 
arrangements are such that user involvement is encouraged. 

Among the external features that can have a bearing on use are 
school -community relations. Specifically, when the local community has a 
stake in the program being evaluated, and when it is asked to provide 
support, then the involvement of community members in the evaluation 
becomes important. 

Finally, certain characteristics of the program itself — including 
its age/maturity, its innovativeness, and its overlap with other programs 

have a bearing on evaluation use. For instance, newer and more 
innovative programs benefit more from formative than from summative 
evaluations. When programs overlap with other projects to form a 
programmatic whole, and when the whole is subject to evaluation, some 
attention should be given to the unique questions and information needs of 
each individual project. 

To summarize the influence of the evaluation setting on use, the 
findings stress the need for Information users and information providers to 
be sensitive to the contextual factors that affect information needs and 
use. For example, in any evaluation setting, the degree of tension or 
conflict between external evaluation requirements and the needs of the 
program itself can have a profound effect on use. Further, within the 
particular organization in which the evaluation is being conducted, overall 
institutional arrangements which minimize program-level autonomy can thwart 
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evaluation use at the program level. Institutional sense of risk, 
community involvement and expectations, the age and innovativeness of the 
program, and its relationship with other programs can also affect the 
extent to which the evaluation is put to use. 

Evaluation characteristics affecting use : The final group of 
characteristics affecting evaluation use relate to the conduct of the 
evaluation itself. They are: the evaluation procedures, the information 
dialogue between evaluator and users, the substance of evaluation 
information, and evaluation reporting. 

Evaluation procedures — including the methods selected, the way in 
which mandated tasks are handled, and reliance on a general model — can 
have d strong bearing on use. In particular, the methods selected should 
be appropriate to the given setting and should satisfy requirements for 
rigor. With respect to rigor and appropriateness, however, if the 
evaluator insists on adherence to strict research procedures, then he or 
she may damage the use potential. Conversely, when the procedures selected 
are sensitive to the purpose and scope of the programs 1 evaluation needs, 
the potential for use is enhanced. 

Earlier, we pointed out that tension may exist among an evaluation's 
various requirements. If the evaluation emphasizes, or is perceived to 
emphasize, externally mandated requirements, then its potential usefulness 
decreases, at least in local eyes, on the other hand, if evaluations are 
designed, or are permitted the flexibility, to meet local needs, the 
potential for utilization increases. 

An evaluation need not follow a formal model to enhance use. What is 
more important is that the evaluator select methods which are sensitive to 
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the organizational characteristics described earlier. This point is of 
particular importance to the use-conscious administrator given the tendency 
of some evaluators to espouse a particular evaluation model regardless of 
its appropriateness in a given setting. 

The importance of evaluator-user dialogue cannot be over-stated. For 
instance, use tends to increase when the evaluator involves users in 
discussions of such specific evaluation concerns as how to read, analyze, 
and make decisions on the basis of evaluation information. Further, the 
earlier the evaluator and program staff meet to discuss possible evaluation 
uses the greater the liklihood that use will occur. 

The substance of evaluation information provided, especially its 
relevance and specificity, is critical to use. Evaluation information 
which focuses on major program concerns is more likely to be used in 
decisionmaking, particularly when its recommendations are helpful. 
Further, important program decisions are typically based on a broad range 
of information needs, and use tends to increase when users are provided 
with information that they see as being specific to these needs. 

The frequency and timing of evaluation reporting contribute to use. 
In this regard, while the effect of a final evaluation report is difficult 
to gauge, frequent and well-focused formative reports, especially when they 
mesh with the timing of program needs, increase the potential for use. 

As for the format of reports and presentations, evaluation use 
increases when the evaluator selects a format which is appropriate to 
potential users and which gives clear answers to their questions. In this 
regard, informal presentations during the course of an evaluation are 
especially effective. Finally, graphic, narrative, and nontechnical modes 
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of presentation ~ especially when they describe program strengths and 
weaknesses and make recommendations for improvement — increase use. 

With respect to the conduct of an evaluation, then, the research 
findings agree that the procedures used, the interaction between evaluator 
and users, the substance of the Information, and the manner in which It Is 
reported can have a marked influence on use. 

Several themes emerge from the preceding discussion. First, the 
evaluator's ability to address questions relevant to the program is a key 
determinant of program-level use. The evaluator 1 s success here depends, In 
part, on the various requirements imposed on the evaluation {by a funding 
agency, for example, or by the program Itself) and on whether any 
particular set of requirements Is allowed to dominate. If one set of 
needs, requirem nts, attitudes, or expectations takes precedence over other 
sets, causing the evaluator to adopt a certain roV and collect certain 
kinds of information, and if the resulting evaluation runs counter to 
expectation *t the program level, then potential users are likely to view 
the evaluator and his or her work with something loss than enthusiasm. 

Second, evaluation use also depends In large measure on users 1 
interest in the program and Its evaluation and on their commitment to 
applying Its findings. This commitment Is, In turn, affected by their 
levels of professional experience, their perceptions of Individual and 
institutional risk, the program's context, and the program's relationship 
with similar programs or projects undergoing evaluation. 

Third, even In those cases where the evaluation addresses the needs 
and concerns of program-lev 1 staff and where potential users are 
predisposed to apply its findings, certain procedural handicaps can offset 
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a bridge between the research findings and their application. The Guide 
concludes with a series of worksheets designed to help the administrator 
promote the use process and monitor Its progress. These worksheets also 
contain suggested strategies the administrator might adopt to stimulate 
use. 

The factor pattern guiding the organizing process and some exemplary 
worksheets and strategies appear In Appendix A. These materials reflect 
the Guide's basic purpose ~ to help an administrator organize an 
evaluation for maximum use. The demonstration project was conceived to 
ascertain how well the Guide met that purpose. 



The Demonstration Project 



As mentioned previously, we view evaluation as a means of producing 
Information to be used In making decisions about educational programs. In 
this regard, the evaluation use literature shows that such use Is 
Influenced by factors reflecting the participants In the evaluation, such 
as the evaluator and potential users; by factors In the evaluation setting , 
such as requirements, organizational features, and program features; and by 
factors stemming from the evaluation Itself, such as Its procedures for 
collecting and communicating Information. If these factors are Ignored, 
they can reduce or even negate an evaluation's use potential. If, on the 
other hand, an evaluation Is planned, conducted, and communicated In 
response to the particular factors operating In a given setting, Its use 
potential can be greatly enhanced. 

The Guide we developed, therefore, provides persuasive Information 
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highlighting the need for someone to assume responsibility for ensuring 
that an evaluation *s conducted in the manner suggested above. More 
importantly, the Guide also provides planning and implementation worksheets 
that the evaluation organizer can use as he or she organizes the evaluatio r 
for use. These practical materials also provide suggestions that the 
organizer can apply or adapt to ensure that the evaluation's intended uses 
are achieved. 

The Guide is designed to be used by a program administrator who 
assumes responsibility for the evaluation's organization, primarily because 
a program administrator is often strategically placed to assume 
responsibility for promoting evaluation use. Of the forty-five or so 
discrete elements identified as affecting evaluation use, only about 
one-third of them stem from the evaluator's personal and professional 
approach to his or craft, while the remaining two-thirds of the identified 
elements reflect users' personal and professional characteristics and the 
organizational setting in which the evaluation is conducted. Elements in 
these user and setting domains are part of the administrator's sphere of 
influence and therefore the administrator should assume responsibility for 
ensuring that the evaluation deals with them in an appropriate manner. 

Tne Guide is intended to facilitate administrator assumption of this 
responsibility. It exemplifies the use-influencing characteristics we felt 
would operate in most settings; offers strategies we hoped would minimize 
negative influences while strengthening positive influences on use; and 
provides a means of tracking progress toward the intended evaluaticn uses. 

The demonstration project set out to determine the effectiveness of 
the Guide at the local program level. The project's objectives were 
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as follows: (1) to Implement the decision maker's Guide in schools and 
districts conducting program evaluations; in order (2) to the assess the 
extent to which our use framewonc would bear up in real evaluation 
settings; (3) to validate the existence of the use-influencing factors 
emphasized in the Guide; and (4) to assess the extent to which the Guide's 
suggested strategies promote evaluation use. 

Site Selection 

This phase of our work, first, sought the advice of practitioners and 
researchers to help us locate and Invite schools districts which might be 
willing to participate in our demonstration efforts. 

With the assistance of these advisors, we located a sample of such 
districts and selected them to ensure that they differed 1n features such 
as size, location, population, program emphasis, and evaluation needs. 
This sampling pool was sufficiently large to ensure that we would attain 
eight to ten broadly representative program sites. 

Next, we conducted exploratory telephone conversations with district 
administrators {usually the superintendent) to discuss the project and 
possible district interest 1n participating. Each administrator who 
Indicated interest received a formal letter of invitation and a description 
of the project (see Appendix B). 

From the twenty formal invitations we extended, ten broadly 
representative districts agreed to participate. Each of these districts 
designated a program administrator who would assume responsibility for 
organizing the evaluation for use, and each of these administrators 
received a copy of the Guide. Since we wanted to ascertain the 
effectiveness of the Guide 1n and of itself, future CSE support consisted 
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Results 

The Program for Gifted and Talented High School Students 

This program is situated In a large district in the midwest. Its goal 
is to provide instructional and motivational support to assist gifted and 
talented sophomores to successfully compete on tests such as the 
Preliminary Scholastic Aptitude/National Merit Scholarship Test 
(PSAT/NMSQT). To meet this goal, the pr *ram provides accelerated, 
in-depth instruction in the areas of languge and mathematics, with emphasis 
on test-taking strategies and motivational techniques. 

The program is funded under Chapter 2 and is required to undergo 
evaluation. The primary goal of the present evaluation was to determine 
whether the program's instructional strategies prepared students to take 
the PSAT/NMSQT. 

The program was implemented in four nigh schools. Students scoring 
above a certain cut-off score on the district's achievement test were 
invited to participate in the program. Some 2,000 students elected to 
participate. 

The director of the district's comprehensive gifted program was the 
evaluation use organizer for this program. The director, who has been a 
district administrator for 17 years, volunteered because previous overall 
program evaluations had not provided information useful for program 
monitoring and improvement. 

More specifically, at the time the evaluation was being considered, 
the principal concern was with whether the student selection process would 
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attract the students who could profit most from the program and, once in 
the program, if the experiences it provided them were effective. 

The program evaluator was a specialist who has been with the district 
for eight years. He had not previously worked with gifted program 
personnel. This evaluator was charged with the task of providing 
information reflecting (1) the effectiveness of the criteria set for 
selecting students for the program and (2) the effectiveness of the program 
developed for these students. 

The intended users of the information addressing these questions were 
staff who administer the program and classroom teachers participating in 
the program. 

The principal sources of data concerning the program consisted of 
standardized achievement tests, questionnaires and interviews for teachers 
and students, interviews with other program personnel, and classroom 
observation. 

Factors requiring administrator attention : Three contextual features 
help explain the manner in which the organizer decided which factors would 
be an influence on the evaluation use process. 

First, previous evaluations of the district's comprehensive gifted 
program had been conducted by independent consultants. Program staff — 
both administrators and teachers ~ felt that these evaluations had 
provided them with virtually no information to help them assess how various 
program segments were operating. In essence, they felt that they had been 
totally excluded from the evaluation process in the past. 

Second, the use organizer felt that, for the evaluation to be useful, 
it would have to focus on the particular gifted program segment of interest 
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rather than provide a broad view of the entire program. In addition, the 
organizer was concerned that the evaluation should provide ongoing 
Information focusing on questions and needs expressed by potential users. 
Based on the users' negative evaluation experiences In the past, the 
organizer knew that It would be difficult to keep his users Interested In 
the evaluation, to convince them of the value of providing periodic 
Information to the evaluator, and to believe that such Information would 
pay off by providing Information of direct use during the life of the 
program. 

Given the above setting, the organizer decided to ask the district's 
research and evaluation unit to conduct a program evaluation. This 
decision points up the third contextual feature of Interest here. The 
district evaluation unit had not previously worked with gifted program 
staff and so any evaluation specialist furnished by the dlstlct might be 
viewed with some suspicion. 

With these relationships In mind, the administrator decided that user 
Interest In the evaluation and their commitment to use would bear 
attention. Similarly, given the need for the evaluator to be In close and 
frequent contact with the program, evaluator commitment to use , willingness 
to involve users, political sensitivity, and credibility would likely 
Influence the use process. 

In addition, because of the currently negative staff disposition to 
evaluation, the organizer was convinced that evaluation procedures , 
Information dialogue, and substance of the evaluation would play an 
Important role in the use orocess, and would need to reflect users' 
information preferences . 
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Use-promoting strategies : The first thing the organizer did was to 
discuss his evaluation needs wfth the head of the district's evaluation 
unit. The upshot of this meeting was the decision to assign one evaluator 
to stay with the program during its duration. The evaluator selected for 
this assignment was carefully chosen to match the organizer's concerns. 
That is, evaluation staff differed in terms of their penchant for working 
closely with program staff. Some staff members appeared to be more likely 
than others to accept the fact that evaluation procedures selected must 
meet users' concerns and needs just as much as they need to meet technical 
criteria. The evaluator finally selected for the program had previously 
demonstrated the ability to work closely with teachers. 

Once the evaluator was selected, the use organizer arranged a series 
of meetings 1n which users expressed their concerns, questions, and 
Information needs to the evaluator. These meetings helped allay user 
concerns about the value of participating in yet another evaluation. It 
also gave them an opportunity to raise evaluation questions and to hear 
that these qustions would be deal* with by the evaluation. In this way, 
their commitment to use began to evolve. 

At the same time, during these meetings the organizer asked the 
evaluator to carefully consider the range of user needs, to propose to 
users alternate means of answering their questions, and to allow users to 
express their opinions as to the relevance and feasibility of various 
measures. In this way, the evaluator was able to demonstrate his 
sensitivity toward program needs and thus establish credibility by 
selecting appropriate measures. 



ERLC 



36 



- 31 - 



During the course of the evaluation, the organizer made sure that the 
evaluator provided the users with ongoing formative information reflecting 
program operations and illuminating questions previously expressed by 
users. In addition, the organizer made sure that the evaluator was readily 
accessible to answer questions, to explain findings, to discuss their 
implications for program operation. 

This communicaton system, consisting of both oral presentations and 
written Interim and final reports, helped maintain user commitment and 
further enhanced the evaluator's credibility. 

Throughout the entire process, the administrator made sure that 
procedures propi d by the evaluator would be accepted by the users. 
Frequently, the administrator advised the evaluator to "soften" his 
approach, to make sure that technical considerations were not seen as more 
important than the real -world constraints of program staff. 

Throughout the entire program, the organizer monitored the evaluation 
communication system. He made sure that users were Invited to review 
results, to critique draft reports, to offer their insights to the 
evaluator. These insights were then considered by the organizer and 
evaluator and were frequently added to the contents of a report. 

Success in achieving the intended uses : We pointed out earlier that 
this evaluation had two principal audiences. These audiences consisted of 
program administrators and classroom teachers. The administrators were 
primarily concerned with whether student selection criteria were 
effective. The teachers were primarily concerned with the effectiveness of 
the program's Instructional strategies. The evaluation use organizer 
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wanted the evaluation to provide information that both user groups could 
apply- 
In terms of student selection criteria, the administrators discovered 
that students 1 prior achievement scores were not the best nor the only 
predictor of their success in the program. That is, interviews with 
students suggested that the strength of the decision to attend college was 
a likely predictor of success. Given this evaluation finding, the program 
administrators now make sure that candidate students, in addition to having 
certain achievement test scores, demonstrate in a one-on-one interview that 
they actually plan to attend college. 

In terms of the effectiveness of the program's instructional 
strategies, the teachers discovered, on the basis of student inteview data, 
that they were taking up too much time in the presentation of test-taking 
strategies and that they should spend more Instructional time dealing with 
language and mathematics. Most students suggested that while some coaching 
in test-taking strategies was useful, they felt that they needed greater 
mastery of the content of tests, such as language arts and mathematics. 
The program for these students has been modified accordingly. 

In the particular setting, then, the strategies adopted by the 
organizer appear to have stimulated the intended evaluation uses. 
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The Emotional Adjustment Program 

This program is located in a medium-sized district on the eastern 
seaboard. It is part of the district's overall special education program, 
and is focused on students identified as having emotional problems which 
affect their achievement in school. 

Funded directly by the district, the program involves 100 students 
fairly evenly distributed 1n an elementary school, a junior high school, 
and a high school. Each student Identified as being in need of the 
program's services receives psychological and educational counseling 
designed to facilitate achievement of the objectives and progress rates 
stipulated 1n his or her individualized education plan. 

While the district's overall special education program had been 
evaluated in the past, this was the first time that the emotional 
adjustment program had been singled out for close attention. It was 
conducted at the request of the program coordinator, who also served as the 
evaluation use organizer. 

The evaluation was conducted by a member of the district's student and 
program evaluation staff. This person has been with the district and 
working as an evaluator for two years. 

As mentioned above, the evaluation was requested by the program 
coordinator. Her request for the evaluation reflected her concern that she 
did not have a clear picture of how the program was being implemented. 
That is, while the program's curriculum guide was a fairly comprehensive 
document, the implementation level was the individual classroon, and so the 
coordinator felt that future plans for the program would profit from 
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descriptive detail at that level. 

The task requested of the evaluator, then, was to conduct an 
implementation evaluation of the progam. The intended primary user of the 
information was the program coordinator herself. She planned to use the 
evaluation as a baseline for future programmatic decisions. 

The intended secondary users of the evaluation were to consist of 
other special education staff whose support might be required in the case 
of programmatic changes, emotional adjustment staff who might be required 
to alter current practices to accomodate these changes, and the board of 
education in matters of continued support of the program. 

The principal sources of data for the evaluation consisted of student 
demographic data, results of teachers' observation of students 1 behavior, 
examination of individual education plans, examination of the program's 
curriculum guide, and evaluator-conducted interviews with participating 
students and staff, and with staff of other special education units. 

Factors requiring adminstration attention : The kinds of evaluation of 
the special education program conducted in the past help explain the 
factors that concerned the organizer here. First, these evaluations had 
taken a broad focus the special education program rather than the 
discrete units comprising the program. Second, these evaluations had a 
marked quantitative flavor — emphasizing the numbers of students served 
rather than judgments of how well the program met their needs. Third, to 
the extent that more qualitative appraisals were made, they stemmed 
primarily from examination of students 1 individual education plans to the 
virtual exclusion of other programatic features. 

In this context, then, the organizer anticipated that user 
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characteristics would play a large role in determining the extent to which 
the eval'^tor would be able to gain access to the finely-grained 
Information required by an Implementation study. Staff had not previously 
been exposed to this conception of evaluation: unless they cooperated with 
the evaluator, the organizer would not get the Information she wanted to 
use for future program pl;nning; unless they accepted the validity of the 
evaluator's findings, they would be unwilling to use the evaluator's 
recommendations If they ran counter to their not ins of the program. 

The organizer decided, then, tht\ user vie , about the program and 
their expectations for the evaluation would be critical. As a corollary, 
it seemed clear that the evaluator's political sensitivity would largely 
determine his credibility . Unless these factors were handled with care, 
che necessary commitment to use would be problematic. 

The orgailzer was also concerned about some of the overall 
institutional arrangements — such as largely autonomous classroom* and 
staff who co n tribute to the program but who are located in organizational 
units outside special education. 

In terms of evaluation procedures, the organizer was concerned with 
whether the Intention to Interview would be seen by staff as an appropriate 
method, and with their likelihood of viewing information which would 
aggregate across classrooms as having rele v ance to them as Individuals. 

Use promoting strategies : The evaluation organizer was concerned 
about the possibility of the various user groups being unwilling to 
participate In the evaluation and apply Its findings. Before introducing 
the evaluator to the staff, she invited members from other special 
education units and staff of the emotional adjustment program to a meeting 
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in which the proposed evaluation was discussed. 

In this meeting it was evident that most members saw evaluation more 
as a means of demonstrating compliance than a vehicle for characterizing 
program operations and identifying areas of strength and weakness. At this 
juncture the organizer suggested that, as it would be applied to the 
program, the evaluation would have an essentially descriptive function, 
that the intended users of its findings were program staff, and that any 
recommendations proposing changes to current practice would be presented 
for staff input before any final decisions were made. 

This characterization of evaluation tempered all use-promoting 
tactics adopted by the organizer and/or the evaluator. For example, at tne 
first meeting between staff and evaluator, the organizer had the 
evaluator amplify and illustrate the formative uses of evaluation. One of 
the off-shoots of this meeting was the formation of an advisory group, 
established by the organizer, who would consult with the evaluator as the 
evaluation evolved. 

Working in close cooperation with the evaluator and organizer, the 
group helped establish feasible time lines for data collection, took 
responsibility for setting up schedules of meetings between evaluator and 
program participants, and generally worked as liaison between the evaluator 
and other potential users. 

Success in achieving the intended uses : Earlier we mentioned that the 
organizer considered herself as primary evaluation user. In this regard, 
the evaluation was a success. For example, the evaluation discovered that 
there was great variation in how staff members viewed the purposes of the 
program and their roles and responsibilities within it. Teachers used 
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various means of observing and recording student behavior. They also 
tended to differ on the kinds of behavior they thought they were supposed 
to be watching for, and In how they attempted to modify student behavior. 

The manner In which Individual education plans were developed was also 
subject to variation. Some staff members tended to rely on a certain kind 
of Information more than others, the plans 1 level of specificity or 
generality varied; means of assessing student progress varied. 

In addition, the evaluation pointed out that staff of the emotional 
adjustment program did not know much about what their colleagues were 
doing. This finding held up in other areas: staff members of other 
special education units did not know a great deal about the functions of 
the adjustment unit and vice versa; staff members who had Information of 
concern to the emotional adjustment program, but who were housed in other 
district units, such as pupil services, were concerned about "turf" and 
Initially unwilling to share Information without a lot of "personal 
coaxing" from the organizer. 

While the evaluator did encounter some problems — having to 
re-schedule appointments, having to accept a sample of respondents that was 
perhaps a little on the snort side, having to give up, to ensure staff 
cooperation, what he saw was a technical requirement In a measure or Its 
administration — the process want fairly smoothly. The use organizer, as 
primary user, did receive a report that pointed out Important features 
about how the program was being Implemented and which contained 
recommendations about how the organizer,, as unit coordinator, might present 
these findings to the other (secondary) Intended users before any 
recommendations for change are seriously contemplated. 
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The Early Childhood Program 

This program is located in a large district in the south-eastern 
United States, Serving children ranging from infancy to kindergarten, the 
program's principal intention is to foster positive attitudes in young 
children for learning. The major characteristics of the program consist of 
individualized instruction based on the results of diagnostic testing, 
small teacher-pupil ratios, and on-going staff development geared toward 
pupil needs. 

The program lias multiple funding sources and its evaluation is 
required by the district. The principal focus of the evaluation is to 
monitor program operation, to assess its strengths and weaknesses, and to 
offer recommendations for modifications, if necessary. 

The program is conducted in approximately a dozen sites varying in the 
age groups of the pupils served. The supervisor of the district's early 
childhood activities assumed the responsibilities of evaluation organizer. 

The program evaluator has been with the district for several years, 
and has been working as an evaluator for the last two years. She and the 
early childhood taff have worked together in the past. 

The intended evaluation users were the principal and teachers at each 
site. The primary sources of data on the program were the results of 
diagnostic testing, evaluator observe ion of classroom operations, and 
interviews with program staff. 

Factors requiring administrator attention : In an early, pre-planning 
meeting between organizer and evaluator, a careful review of the factors 
that might affect the evaluation led to the decision to concentrate 
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primarily, though not exclusively, on the evaluation's procedures. 

Since the evaluator and staff had previously worked together, the 
organizer felt that the usual factors of concern stemmming from evaluator 
characteristics would not be an issue of concern. 

Similarly, given the relative freedom from bureaucratic constraints, 
the organizer felt that organizational features were not likely to become a 
concern. On the other hand, given the innovative nature of the program, 
program characteristics might require the organizer's attention, as might 
such user characteristics as professional style and information preference . 

With factors reflecting the participants and the setting focused as 
outlined above, the organizer (and the evaluator) were therefore in the 
position to devote a great deal of their time and energy on factors 
reflecting the conduct of the evaluation Given the principal's and 
teachers 1 (the ultimate users) heavy involvement In and professional 
commitment to the progrdm, the organizer was convinced that constant 
dialogue , 1n a variety of forms, would be the most Important contribution 
to the use process. Further, such dialogue would begin before the 
evaluation began, continue dunng Its execution, and play a large role In 
matters of reporting. In this way, one particular factor — dialogue 
would be used to make determinations abut evaluation procedures , substance 
of its information, and how that information would be reported . 

Use-promoting strategies : As mentioned previously, on-going staff 
inservice 1s a large component in the program. In that program teachers 
and their principals were already accustomed to frequent sessions in which 
most program staff were present (sometimes with participation cutting 
across all sites and sometimes confined to one site), the organizer 
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inserted "evaluation" as a topic in most, if not all, staff inservice 
sessions. In this way, participant involvement in the evaluation became a 
natural part of their professional responsibility and development. As the 
organizer put it, staff (users) were involved in "preparing for the 
evaluation, doing the evaluation, monitoring the evaluation, and how it 
would be used." 

The organizer described this use-enhancing process as follows: 
The organizer and evaluator prepared a draft proposal describing how 
the evaluation might be conducted. Users 1 review and critique of this 
draft allowed them to insert questions of interest to them that the 
evaluation should address, as well as to make suggestions about how these 
questions might best be assessed. 

Using the above kinds of feedback, the organizer and evaluator 
modified the proposal, built in users questions, and developed an overall 
eva «n plan — questions to be addressed, the measures to be used, data 
collection methods and timelines, reporting schedules and manner of 
reporting. 

User review of the plan led to some refinement and the evaluation got 
underway. 

As data began to be collected, and with the time schedule as a 
referent, organizer and evaluator sought input from the users as to 
reporting formats, presentation of data, and time(s) at which information 
would be needed to allow users to respond to recommendations. 

In this way, a natural and on-going two-way dialogue guided the 
evaluation from its inception, through its progress reports, which at 
minimum were presented and critiqued during staff inservice sessions, all 
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the way to the final report. 

Success In achieving the Intended uses : In terms of its intended 
uses, this evaluation was extremely successful. Curriculum areas in need 
of improvement were identified; principals and teachers responded favorably 
to the findings and changes to the curriculum have been made. 

The diagnostic testing component ~ a critical feature of the program 
— was found to stand in need of improvement, in terms both of content and 
administration. This program componenent was overhauled and staff have 
received inservice focusing on the use of the new measures. 

In this setting, it appears that the pre-established credibility of 
the evaluator, the autonomy of program staff in terms of planning and 
directing their own evaluation, and evaluator receptivity to ongoing 
intimacy with program staff, freed up all parties to concentrate almost 
exclusively on factors reflecting the co nduct of the evaluation . This 
freedom led to a highly used evaluation. 
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The In-School Suspension Learning Program 

This program is located in a large, midwestern school district. It is 
one of several, coordinated district-wide activities intended to help 
reduce the drop-out rate of high school students. A perceived need both to 
improve and increase student attendance and to enforce the district's 
discipline code led to the conception of In-school suspension learning 
centers* 

The program is designed to provide counseling and tutorial services 
for high school students who would otherwise be suspended from school. The 
learning center is a separate classroom where students meet with teachers 
and other school personnel to explore and determine remediation activities 
for misconduct. For students spending two or more periods In the center, 
the program provides opportunities to wor!: on assigned activities related 
to their regular classroom Instruction. Students may remain in the center 
for the duration of their suspension. A teacher specialist and a school 
aide coordinate the center's activities and direct Individual learning 
activities for the students. 

Students participating in the suspension program are selected and 
assigned by the local school principal or his designee who also determines 
the extent* * m the services for each student. 

'.ii. program Is funded under Chapter 1 and is required to undergo 
evaluation. The major goal of the evaluation was to determine the extent 
to which the program helped alleviate the problem of students dropping out 
of school because of behavior problems. In this regard, evaluation data 
were Intended to help make decisions concerning program Improvement. 
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As mentioned above, this program is one of several Intended to reduce 
the student drop out rate. The executive director of this larger effort 
volunteered to be the use-organizer for the 1n-school learning centers 
evaluation. His principal concern as organizer was to ensure that the 
evaluation would be used by program planners, administrators, and 
instructional staff to respond to program strengths and weaknesses 
identified during the evaluation. 

Ttv? program evaluator was a district specialist who has been with the 
district for 16 years, the last two of which as an evaluator. She had not 
previously worked with staff of the learning centers. 

The primary sources of data on the program consisted of monthly 
reports of student participation, on-site observations of learning centers, 
observation of staff development meetings, and questionnaires for staff and 
students. 

Factors requiring administrator attention : The factors which emerged 
as requiring administrator attention stemmed from four principal sources. 
First, the project was newly adopted and undergoing Its first year of 
Implementation. Second, the program operators had designed and Implemented 
the program which they were now conducting. Third, most of the teachers 
Involved in the program viewed evaluation as something required, something 
done to demonstrate compliance rather than to provide Information to be 
used -n program monitoring and refinement. Fourth, the program staff and 
evaluator were working together for the first time. 

Given these contextual considerations, the use organizer had several 
major concerns. For example, he was concerned that the Introduction of an 
unknown evaluator, coupled with the negative disposition some staff felt 
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toward evaluation In general, would reduce or negate the evaluation's ur . 
potential. Further, while a few sta^f members seemed to have a more 
positive perception of evaluation, they still felt that the program was not 
ready for evaluation given Its recent adoption. Finally, since the staff 
operating the program had been Intimately Involved in Its conception, 
planning, and development, the organizer was concerned that they might not 
welcome findings running counter to their judgment of the program. 

With this background In mind, the organizer felt he would have to pay 
particularly close attention, first of all, to the evalua tor's 
credibility . At the same time, he was concerned that Introduction of the 
evaluator and establishing her credibility should also attempt to generate 
credibility for evaluation itself; that Is, elements Influencing user 
Interest In th» evaluation , such as their views of the nrojeci:, their 
expectations for and predisposition toward evaluation, and their needs and 
perceived risks seemed to have a high potential for hampering the 
evaluation 's use. 

Since the organizer perceived no organizational constraints on his 
freedom to direct the course of the evaluation, his final concern was that 
the procedures of the evaluation should assume a formative posture, with 
on-going dialogue and frequent sharing of information . 

Use-promoting strategies : In his first meeting with the evaluator, 
the organizer discussed with her the above kinds of Issues. They both 
agreed that meaningful staff Involvement In planning and conducting the 
evaluation would likely pay off In terms of promoting possible Information 
use. 

At the first of several meetings Involving staff and evaluators, the 
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organizer highlighted the evaluator's background as a teacher and project 
coordinator before moving into evaluation. At this meeting the organizer 
had the evaluator point out the purposes of formative evaluation and 
discuss how it can address staff-generated questions. 

The organizer next set up a series of meetings in which staff proposed 
questions for the evaluation and in which evaluator and staff jointly 
planned and developed measures for these questions. 

In addition to establishing a time frame to guide the evaluator's 
reporting to the staff, the organizer also had her attend regularly 
scheduled staff inservice sessions. The major function of these sessions 
was for the evaluator to elaborate her evolving perception of how the 
program was proceeding, to point out areas seemingly in need of refinement, 
and to receive staff feedback as to the accuracy of her perceptions and the 
relevance and feasibility of her recommendations. 

Over time, staff came to accept the value of the information they were 
receiving: that is, they began to see that evaluation need be neither 
"window dressing" nor "hatchet grinding;" they began to look forward to 
periodic reports — verbal and written — and the opportunity to respond to 
the organizer's invitation to critique and even suggest modifications to 
these "drafts." In large measure, the final report had been negotiated 
throughout the entire course of the evaluation. 

Success in achieving the intended uses : The evaluation seems to have 
a strong potential for helping inprove student attendance and discipline. 
For example, the evaluator's formative reporting helped identify areas in 
which staff needed inservice. It also suggested that the learning centers 
need to enlist community support of the program. Had the staff not been so 
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intimately involved in the evaluation it is unlikely that they would have 
entertained such recommendations. 

The organizer has reported these kinds of findings to his superiors. 
He feels that if staff continue to help identify the skills they need in 
order to work with this particular student population, snd if the immunity 
demonstrates their support of the program to their children, tho program 
has a good chance of being successful. 
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The Bilingual Program 

This program, situated in a small district in the south, is geared 
toward the needs of limited English-proficient (LEP) students. The 
program, which emphasizes language arts, reading, and math skills 
development, is essentially a transitional program. That is, the Program 
allows students to adjust to school and/or master subject matter until 
skill in English is developed to the point that it alone can be used as the 
medium of instruction. 

The program is funded under Title VII and operates in three of the 
district's five elementary schools. Each of these schools has an 
enrollment of approximately five hundred students. In each school, the 
bilingual program serves about 200 students in grades K-4. The vast 
majority of these students are Hispanic. 

In each of the schools with a bilingual program, the principal goal is 
to help students attain sufficient competency in English to ensure 
proficiency in all areas of academic concern. The program evaluation is a 
federal requirement. 

The district does not have a bilingual department as such. The 
bilingual program, and other programs receiving federal funds, is under the 
nominal control of the assistant superintendent. In the past, this 
administrator was given the responsibility for "evaluating" the bilingual 
program. This evaluation had consisted essentially of administering a 
standardized test to students in the program and comparing their scores 
with normative data accompanying the test. 

While these test data, considered alone, suggested that the program 
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was doing fairly well, 5th and 6th grade teachers at each elementary school 
complained that not all students were equally prepared to receive 
instruction in English. This problem had been growing during the program's 
three years of operation. 

The assistant superintendent, concerned with this problem, asked for 
and received permission from the superintendent to hire an independent 
consultant to conduct a careful evaluation of the program. The consultant 
chosen was a faculty member at a neighboring university. 

Assuming the role of the evaluation use organizer, the assistant 
superintendent asked the consultant to conduct an evaluation that would 
help illuminate the problem of variable student preparation to receive 
English-language instruction in the upper elementary grades. The organizer 
intended to use the evaluator's report to make recommendations concerning 
future program operations. 

Given the need to report findings to the federal government, the 
organizer felt that the standardized test in use should continue to be a 
part of the evaluation. The evaluator agreed, but stressed the need to 
apply other measures, such as student performance on tests of entering 
language ability, observations of classroom operation, and interviews with 
students, teachers, and principals. The organizer believed that this kind 
of data would help answer his questions, but was concerned about how staff 
might respond. 

Factors requiring administration atcention : Since the program had not 
been subjected to this kind of evaluation scrutiny in the past, the 
organizer was extremely concerned about the equator's credibility . In 
addition, he was concerned that the evaluator would need to generate 
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rapport with starf in order to enlist their support of the evaluation. The 
extent to which the eva'.jator might positively involve staff in the 
evaluation might be a principal key to suc-ess. 

At the same time, the organizer was concerned about staff Timber's 
professional experience, their predisposition to evaluation, and the kinds 
of risks they might associate with the evaluation. Should these factors 
take a negative turn, staff commitment to use the evaluation — 

Icul'rly in the case of recommendations for > .nge In current practice 
..ngnt be difficult to achieve. 

In addlt-on, given the relative automony of schools and classrooms, 
the organizer was concerned that the evaluator might have difficulty 
gaining access to the Information he would need. Similarly, he had some 
misgivings about using observation and interview methods , and how the 
evaluator would present findings. 

Use-promoting strategies : The first step the organizer Uok was to 
provide the evaluator with a sense of how staff might feel about the new 
approach to evaluation. Some staff had several years of teaching 
experience; others were quite new to the district and might feel threatened 
by the evaluation; none had ever had a very Intimate contact with 
evaluation and might not see how Its findings could be of relevance to them 
as teachers. 

During this conversation, the organizer asked the evaluatcr If it 
would be possible for the evaluation to contain two distinct trands: one 
strand would consist primarily of reporting the results of staid, ai.'.ed 
testing, as well as the broad findings from other measures, to the funding 
agency; the oU. would consist of a separate, highly-focused report(s) to 
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the organizer and the individual schools offering reasons for why some 
students seemed to be less prepared for English-language instruction 
than others. The evaluator accepted this assignment. 

Having defined roles and established possible trouble points among 
staff, the organizer set up a meeting in which the evaluator discussed the 
approach to evaluation outlined above, pointing out that one evaluation 
concern would be to take a close look at how the program appeared to be 
operating in each school , while the other would be to assess how the 
district , as a whole, was serving the LEP sfdents in tie program. 

At this meeting, staff discussed their concerns about whether such an 
approach to evaluation would take up a lot of their time — i.e., beini 
observed, answering questions in an interview, and so forth. The organizer 
promised the staff that he would keep demands on staff time as minimal as 
possible. 

After this meeting, the organizer, with input from the evaluator, then 
prepared an over-all data collection plan: the issues the evaluator would 
concern himself with while observing classes; the kinds of questions to be 
probed during interviews; the time-line for visiting schools and classes. 

Once this draft plan was proposed, the organizer arranged a meeting 
with each of the three schools. At each school, organizer and evaluator 
presented the plan outlined above, took questions from staff, and agreed to 
some modifications in timing to make data collection more convenient. The 
principal and some teachers at 'ich school asked if they could see reports 
,efore "they were sent to the Feds". The organizer (and evaluator) agreed. 

Success in achieving intended uses ; The organizer ultimately received 
an evaluation report that shed a great deal of light on hi; question about 
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student ability to achieve in all-English-language instructon. However, 
tne evaluator had to spend much more time than anticipated to collect the 
data: teachers asked for postponements for interviews they were scheduled 
for; students who were to be interviewed could not be removed from the 
classroom on the appointed day; information on students 1 entering language 
ability was hard to find; repeated evaluator visits were necessary. 

Eventually, however, the evaluator was able to prepare a report 
containing findings that illuminated the central question. He found, for 
example, that individual teachers varied greatly in the extent to which 
they used diognostic tes;s of entering language ability to guide 
instruction; they varied quite a bit in how they "practiced" with their 
students in preparation for upcoming tests; there was a great deal of 
variation in how they presented instruction and in how they attempted to 
introduce English as the primary language of instruction; exit criteria 
were poorly defined. 

These findings suggested to the organizer that the differing levels of 
student readiness to receive instruction in English, then, were primarily 
an off-shoot of classroom operation and not a reflection of any inherent 
weakness in hew the district planned for students to progress. 

However, the findings did convey a sense that the program was being 
poorly managed at the district and school levels, and that such management 
might be an outgrowth of differing perceptions of bilingual education and 
what it is supposed to accomplish. In chis final regard, the evaluation 
was extremely effective in that it is presently being used by district and 
school personnel to discuss issues of bilingual education intentions and 
practices, the extent to which there 1s, or needs to be, consensus on these 
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issues, and how answers to these questions might shape future bilingual 
education practice in the district. 
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Implementation Issues 

It seems reasonable to conclude that the evaluation-user Guide was 
applied successfully at each of the five sites described above, and that it 
led to evaluations that were put to their intended uses. It seems equally 
clear that, as we expected, . jch success demands the presence of an 
administrator who takes an active role in the promotion of evaluation use, 
coupled with an evaluator who welcomes, or at least accepts, this kind of 
administrative oversight. 

The negative experiences in our five other sites bear out the truth of 
the above observations. In each of these sites the Guide was not put to 
its intended uses. In two of these sites, budgetary reductions announced 
mid-way in the school year demanded an immediate district responbe. In 
each site the person responsible for dealing with the budget crisis was 
also the evaluation use organizer. When these individuals removed 
themselves from active supervision of the evaluation, the matter of use 
essentially became a matter of complying, with as minimal an effort as 
possible, with the bare-bones a the evaluation's requirements. 

The evaluators in these sites, however, cannot be faulted. In each 
case, funds thought to be available for the evaluation were cut drastically 
and, in addition, staff thought to K e available to help generate and 
ultimately put to use evaluation information were shunted off in other 
directions in response to the matter of the budget. 

In two other sites, essentially the same problem arose, although for 
different reasons. In these two sites, the designated evaluation organizer 
left his or her district to assume another position. In each case, the 
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shoes of the departing organizer were never filled and in each case 
the evaluator was instructed to "get back to business as usual". 

In the final site, political problems Interfered with the Guide's 
application. Here the organizer asked for, and received, CSE permission to 
use the guide to organize a teacher-evaluation system. Teacher-union 
opposition to the proposed system precluded the Guide's ever being put to 
use. 
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Synthesis and Impactions 

In those sites where a local administrator assumed responsibility for 
organizing an evaluation for use, and applied *.ne kinds of tactics to the 
core group of factors we suggest in the Guide as being critical, the 
evaluation information was puc to its intei led uses. 

Some of our demonstration sites encountered unexpected occurrences 
leading to the evaluation organizer being removed, part-way through the use 
process, from the evaluation. In these sites the evaluations were not put 
the their intended uses. 

It seems possible, then, that the influence of the organizer on the 
use process is even more important than we conceived it to be. In 
addition, the possibility of the loss of the use-organizer is always 
present. Perhaps we need to add another element to the use process, one 
that beefs up the role of the evaluator. That is, the organizer should 
probably try to draw up an over-all use plan, with relevant factors and 
their associated use strategies spelled out, in extremely close cooperation 
with the evaluator. Together, they should try to anticipate the effect on 
the use process of the organizer's removal from the setting, brain-storm 
factors not anticipated which may stem from this removal, and plan 
strategies the evaluator might apply to offset the negative effect. The 
evaluator, in essence, should be prepared to assume the role of the use 
organizer. 

The elements we lay out in the Guide's core factors pattern clearly 
provide the organizer with an effective vantage point from which to begin 
the use process. Further, in each of our sites, common factors emerged 
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which suggest the extent to which the factor pattern, as a total entity, is 
likely to apply in most settings. For example, most of the 
participant factors, evaluator and user, see* to have a strong -nance of 
affecting use in most settings. Many of the evaluation-conduct factors, 
similarly, would seem to be likely candidates to crop up in most settings. 

However, the extent to which these factors emerge and the strength of 
their influence on use may be much more situation specific than we 
anticipated. That is, none of our successful demonstration site organizers 
either expected, or found it necessary, to plan for and exert influence 
upon al]_ of the participant and evaluation factors in the Guide's core 
pattern. 

Perhaps we should suggest more strongly, then, that the local 
organizer needs to spend more time characterizing the dynamics of his or 
her own evaluation context than we initially thought. Preparing this kind 
of analysis might help focus the evaluation more closely on local issues 
and minimize the possibility of time being spent on non-relevant factors. 

With respect to factor relevance, further, it seems possible that we 
may need to take a closer look at some of the factors we place in the 
setting of the evaluation. For example, in most of our sites there was 
little perceived need to anticipate or react to many setting factors, 
whether of preexisting bounds, organization features, or the project 
itself. We had more or less expected this phenomenon, and so the factor 
pattern and the guide do not place a great deal of emphasis on s'.^h 
factors. 

On the other hand, some institutiona l features of the setting, but 
highly dependent on the particular setting, once again, may have the 
potential to play a stronger role in the use process then we had thought. 
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For example, in one of our cases, the business of "turf and territory" 
had a strong role in the use process, possibly suggesting that 
institutional arrangements may need further consideration. Similarly, 
while we recognize the influence of fiscal constraints on use, we had not 
anticipated that, by itself, this factor would be likely tc have a great 
effect on use* Depending on the particular setting, as we s&w in two of 
our unsuccessful sites, this factor may have an important role to olay. 

Across each of our successful sites, evaluator accommodation to 
program needs is a constant theme. Organizers were careful to ensure that 
the overall evaluation plan and its specific measures paid just as muc, 
attention to the constraints of the setting as they did to methodological 
elegance and technical considerations. In each case, the evaluator was 
willing to achieve this balance and in each case it made an important 
contribution to use. Had the evaluator tried to make the program conform 
to strict notions of design and measurement it is likely that the 
evaluation's use potential would have diminished. 

While the Guide pays attention to this issue, perhaps we should more 
strongly emphasize to organizers and evaluators that "personal" 
considerations play a much greater role in the use process than "technical" 
considerations. 

To conclude then, it appears that the Guide, its core factor pattern, 
and its suggested strategies can be used to bring about evaluation use. 
The principal advice we would offer are for the organizer to closely tailor 
the core factor pattern to the "cut" of his or her own context, to prepare 
the evaluator for the possibility of assuming the organizing role, and to 
reiterate to each the importance of personal demeanor in the promotion of 
evaluation use. 
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APPENDIX A 



Factor Pattern, Worksheets, and Organizing 
Strategies from the Guide 
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Figure 2 

Factor Pattern for Evaluation Un 

A. Setting the Stage 

Pre-existing e valuation bounds 
User identit y 
Project charac teristics 
Organizational features 



l - Identi^rfng/Organiang the Participants 
User characteristics: 

* interest in evaluation 

* commitment to use 

* professio nal style 

E valuator characteristics: 

* background and identity 

* credibility 

* choice of role 

* willingness to involve users 

* rapport with users 

* commitment to use 
* politica l sensitivity 

Evaluation procedures (plan) 



C - Operationaliang the In teractive Process 
Evaluation proced ures (execution) 
Substance of evaluatio n information 
Evaluator commitment to use 
Informa tion dialogue (formative) 
User information proces sing preferences 

D - Adding the Fini shing Touches 

Evaluation reporting 

Evaluator commitment to use 
Evaluator political sensitivity 

Information dialog ue (summative) 

User commitment to use 

i ■ 



0 

ERIC 



66 



Organizing for Evaluation Use 



WOrkSheet A: Setting the Stage 



Evaluation Topic: 



Cluster A: 


Pre-existing evaluation bounds 






User identity 






Project characteristics 






Organizational features 





With respect to my own program and the above topi-, I need to keep the following in mind: 



Factor/Element 
Affecting Use: 

A-1 Pre-existing 
evaluation bounds 
(written requirements; 
other contractual 
obligations, fiscal 
constraints) 



Relevant Information: 

1. 



Things I Can Do To Influence Use: 

1. 



2. 



4. 



4. 



5 



5. 



Organizing for Evaluation Use 

Worksheet 



Figure 4 

Setting the Stage 



Evaluation Topic: Should SABER be continued as a 
district-funded activity? 



Cluster A: 



Pre-existing evaluation bounds 



User identity 



Project characteristics 



Organizational features 



With respect to my own program and the above topic, 1 need to keep the following in mind: 



Factor/Element 
Affecting Use: 

A-1. Pre-existing 
evaluation bounds 
(written requirements, 
other contractual 
obligations; fiscal 
constraints) 



Relevant Information: 

1. Federal funding will cease at the 
end of the current school year; 
the Board must make a decision 
in the Spring whether to continue 
the program 



2. Federal requirements mandate the 
use of norm-referenced acKsvement 
tests 

3. In addition to achievement data, 
the Board will need the following 
types of information to inform 
its decision: student enrollment 
data, staffing needs and responsi- 
bilities; cost data; previous 
evaluation reports. 



Ten percent of the SABER budget 
is allocated for evaluation 
activities. 



Both the Board and the federal 
funding agency want more explana- 
tion of the discrepancies in the 
first two years' testing results. 



Things I Can Do To Influence Use: 

1 . Plan a timeline for all evaluation 
activities so that the required 
information will be available to 
the School Board when needed. 
Clarify the feasibility of this 
timeline with the evaluator. 

2. Review test administration and 
student preparation for test-taking 
with approp' ; ate staff members. 

3. Make sure all data areas are 
covered in the current evaluation 
plan. Review past data applica- 
bility to current Board information 
needs; have the evaluator .econstruct 
any missing data pieces 

4. Make sure enough testing supplies 
are on hanu, investigate less 
costly scoring services. 

5- Confer with the evaluator to identify 
anv possible inconsistencies in test 
administration and/or analyses in 
the first two years that may explain 
discrepancies in results. In 
addition to examining these proce- 
dural fbC*ors t apprise the evaluator 
of the nee<f to identify any 
programmatic variables that may be 
responsible for the observed 
tfiscrepancieti. 
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APPENDIX B 



Letter of Invitation 
and Project DescriptTon 



UNIVERSITY OF CALIFORNIA, LOS ANGELES 



Los Asr I ts 



SA\ DIK.O 




UCLA 



SaSTABAHBAHA - SASTACRLZ 



CENTER FOR THE STUDY OF EVALUATION 
UCLA GRADUATE SCHOOL OF EDUCATION 
LOS ANGELES, CALIFORNIA 90024 



Oear : 

Thank you for your interest in participating in CSE's project on 

administrative organizing for evaluation use. As I mentioned on the phone, 

all of the research on evaluation use agrees that in order to build locally 
relevant uses into a program's evaluation, a local administrator (e.g., a 

project director) needs to decide what these us*s should be and organize 
the evaluation around them. 

I've enclosed a CSE report describing ojr research on evaluation use, 
the factors known to affect use, and the process of organizing for use. 
The report introduces the product ~ Organizing for Evaluation Use: A 
Handbook for Administrators — that your designated administrator-organizer 
will receive from us and follow as he or she works with the program 
evaluator to plan for and follow up on uses for the evaluation. 

I've also included an outline describing our project's overall 
structure and the role of the schools or districts working with us. CSE's 
role is not to influerice how you evaluate your pr , grams ^ut rather to fv ~p 
ensure that ther? evaluations have a high use potential. To help with 
this, in addition to the Handbook, CSE staff will be available for tele- 
phone consultation. 

We will ask participating schools and districts to help us document 
the use process. This will primarily consist of filling out an occasional 
(short) questionnaire and some tclephc.e follow up, and asking your 
administrator-organizer to maintain a record of correspondence, meetings, 
and so forth, between him/herself and the evaluator. This documentation 
will also provide some information about the project director and the 
evaluator, and describe the project being evaluated and its overall 
contc-xt. 
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Please let me know as soon as possible if you and your district will 
work with -us in this project, so I can send you the Handbook in time for 
your administrator-organizer to become familiar with its content before the 
school year begins. 

Again, thank you for your interest. I hope we can work together in 
this project and jointly offer an important contribution to improved 
evaluation practice. 

Cordially, 



James Burry 

Senior Research Associate 

JB:klf 
End . 
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ORGANIZING FOR EVALUATION USE 



CSE has been conducting research on evaluation use for the past several 

years. This NIE-funded work has uncovered a variety of factors that affect the 

degree to which program evaluations and the information they provide are used 
for local program needs. 

One of the central findings pinpoints the need for a local administrator 
(e.g., a superintendent; a program director) to share with the evaluator the 
responsibility for planning locally-important uses for the evaluation, and for 
organizing the evaluation 1n ways that help ensure that these planned uses have 
a strong chance of taking place. Without this local leadership, it 1s extremely 
difficult, and often Impossible, to make local use happen. 

To meet that adml n1 strati ve-eva'!uat1ve ne«d, CSE has developed a product 
called Organizing for Evaluation Use: A Handbook for Administrators . This 
handbook summarizes the relevant evaluation-use research findings; describes the 
factors found to have an effect on evaluation use; exemplifies these factors at 
work in a variety of educational settings; and shows how an administrator might 
influence the factors to help promote evaluation uses. 

With these examples as a constant referent, the local administrator using 
the Handbook decides on the uses/users for the evaluation, decides which of the 
factors are likely to have an influence on use 1n his/her setting, and then 
follows a series of step-by-step procedures, with planning and worksheets for 
each, to make sure that these factors have a strong chance of working for the 
intended uses. These organizing procedt es begin while the evaluation is being 
planned and continue throughout the program's evaluation. 

NIE has asked CSE to Implement the Handbook in districts throughout the 
U.S. to demonstrate this evaluation-use process. We are selecting districts 
which: 

° are doing a program evaluation, 1984-1985; and 

° will designate a local administrator to be responsible for working with 
the evaluator to organize the evaluation for use around locally-relevant 
needs such as program planning and improvement; curriculum evaluation; 
staff development; etc. 

We are selecting districts that vary in size and geographic location; we 
are selecting some sites whose "valuator 1s a regular dlstrlct/schoo 1 employee 
and others whose evaluator 1s hired externally; we are Interested 1n having the 
Handbook used in both externally and locally-funded programs. 

For each school or district participating, CSE supplies the Handbook and 
makes staff available for telephone consultation. 

Each participating school or district will be asked to supply CSE with 
information (via short questionnaires; telephone follow-up) to help us document 
the use process at each site. 

Project period is from September 1984 to August 1985. 

James Burry, Senior Research Associate 
(213) 206-1508 
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APPENDIX C 



Questionnaires for Evaluation 
Organizer and Evaluator 
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INTERVIEW: EVALUATION USE ORGANIZER 
A. Personal Information 

the^strlct 9 *" ^ getting some back 9round information on your role in 
1 First, what is your title? 



2. In that position, what are your general responsibilities? 



3. How long have you been in this position in the district? 

4. And were you in any other positions in this district, administra- 
tive or other, before this assignment? 

If yes, what were they? 



5. Have you held positions in any other schools or districts before 

this one? 

If yes, what were they? 



6. In all, then you've been in education for how many years? 

And of that time, you've been an administrator for how long? 

7. Now, in terms of this specific project, did you make the decision 
yourself to be the evaluation use organizer? 

a. If yes, why? 

b. If no, can you give me an idea of how and why you were 
designated as organizer? 
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B. Project Information 

Mow I'd like to cover a few things about the project being 
evaluated. 3 

1. First, does the project have a name? 

2. And what 1s Its main funding source? 

3. Can you give me an Idea of the total funding level? 

4. And how much of that 1s earmarked for the evaluation? 

5. Can you give me a picture of what the project Is trying to 
accomplish? For example: 

a. Who are the principal project participants, such as students, 
staff, other? 



b. For each participant group, can you briefly describe their 
composition, such as grade level(s), language minority students, 
AFDC students, payed teacher aides, and so forth. 



c. For each participant group, what are the project's major goals? 



d. And with respect specifically to the students participating, 
the major content area(s) and goals are: 
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6. Finally, here, can you let me have any material that describes the 
project? 



Context Information 

Let me switch topics now to the broad administrative and 
organizational setting of the project. 

1. First, I'd like to get a sense of how the project fits in with 
other district efforts. 

a. For example, is the project a part of a larger programmatic 
effort? 

b. If yes, can you describe that for me? 



2. Can you give me an idea of the reason(s) for why the project is 
being evaluated? 



3. And wh?t was the reasoning behind choosing this particular project 
to organize its evaluation for use? 



4. Finally, as the evaluation use organizer, what kind of authority do 
you have in trying to set the focus of the evaluation: for 
example, in trying to have it address project/district/school needs 
as well as other evaluation requirements? 



4 



D. Evaluation Procedures 

Let me move now to a few questions about how the project is belnq 
evaluated. 3 

1. Can you tell me a little about the evaluation procedures? 
a. For example, what measures are being used? 



b. For whom? 

c. And how often will each be administered? 



2. Next, who will actually administer these measures? 



3. And who are the people who will get the results of these measures? 



. Can you think of anything else interesting about the evaluation, 
such as the model or philosophy it follows? 



5. '.o you have any material you can share with me that describes the 
evaluation's requirements, purposes, and so forth? 
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E. Evaluation Purposes 

Let me close now with a question or two about low you see the 
purposes of the evaluation. 

1. First, what are your thoughts about what the evaluation should try 
to accomplish? 



2. Next, when you think about the kinds of information the evaluation 
should provide, what's your sense of the people who could use that 
information, and how? 



3. And lastly, what are your thoughts abou. -at you might need to do 
to get that, evaluation use(s) to happen? 
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INTERVIEW: PROJECT EVALUATOR 

Personal Information 

Let me begin by getting some background information on your role in 
the project. 

1. First, what is your title? 

2. And are you a regular district employee, or were you brought in as 
a consultant to be the project e valuator? 

3. I'd like to get an idea now of your background in education, 

a. First, how long have you worked in education? 

b. And in what capacity? 

c. And how long have you worked as an evaluator? 

d. Now, how did you come to be an evaluator? That is, can you 
give ,.ie an idea of your background in evaluation? 



4. And how did you come to be the evaluator for this particular 

project? For example, were you assigned by the district? By the 
company you work for? Other? 



5. Finally, is this the first time you've worked with the people in 

the project, or have you worked with them before? 

a. If yes, in what capacity? 



Project Information 

Let me switch now to one or two broad questions about the project. 
1. For example, can ycu give me an idea of what the project is trying 
to accomplish, and for whom? 
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2. Now, is there anything about the project itseif, such as the 
participants, project goals, and so forth, that presents any 
problems for you as the evaluator? 
a. If yes, can you describe them for me? 



b. How do you think you might be able to overcome them? 



3. In terms of the project's evaluation needs , such as external 

requirements, local evaluation questions, do you have any problems 

in trying to meet them? 

a. If yes, can you give me an idea of what they are? 



b. And how you think you might be able to deal with them? 



4. Following up on the issues you've described to me, is there any- 
thing in the project's context or setting , such as administrative 
or organizational issues, that presents you with problems as 

evaTuator? 

a. If yes, can you describe them for me? 



b. And how do you think you might be able to overcome them? 
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Evaluation Procedures 

Let me move on, now, to a question or two about how you're 
conducting the evaluation. 

1. First, are you following any particular model or evaluation 
approach? 

a. If yes, can you describe it for me? 

2. Next, can you give me a general picture of your evaluation proce- 
dures, such as measures you're using, with whom, and how often? 



3. And finally, can you give me an outline of what you see the 
evaluation accomplishing, and for whom? 
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